feature: add plugin smoke validation workflow#715
feature: add plugin smoke validation workflow#715zouyonghe wants to merge 1 commit intoAstrBotDevs:mainfrom
Conversation
zouyonghe
changed the title
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a plugin validation script and a corresponding test suite to automate the verification of AstrBot plugins. The script includes functionality for repository cloning, metadata validation, and a worker-based loading check. The review feedback focuses on enhancing the robustness of the validation logic, specifically by improving the fallback YAML parser to handle trailing comments, ensuring metadata fields like 'name' and 'version' are correctly handled when provided as numeric types, and making the worker output parsing more resilient to extraneous log messages.
| if not line or line.startswith("#") or ":" not in line: | ||
| continue | ||
| key, value = line.split(":", 1) | ||
| result[key.strip()] = value.strip().strip("\"'") |
There was a problem hiding this comment.
The fallback YAML parser does not handle trailing comments. If a line is name: my_plugin # comment, the value will include the comment text, which will likely cause file-not-found errors later when searching for the entrypoint. Consider stripping comments before parsing the value.
| result[key.strip()] = value.strip().strip("\"'") | |
| result[key.strip()] = value.split("#", 1)[0].strip().strip("\"'") |
| missing = [ | ||
| field | ||
| for field in REQUIRED_METADATA_FIELDS | ||
| if not isinstance(metadata.get(field), str) or not metadata[field].strip() |
There was a problem hiding this comment.
The check isinstance(metadata.get(field), str) will fail if a field like version is provided as a number in YAML (e.g., version: 1.0) and parsed by PyYAML as a float or int. It is safer to allow numeric types and use str() before calling .strip().
| if not isinstance(metadata.get(field), str) or not metadata[field].strip() | |
| if (val := metadata.get(field)) is None or not str(val).strip() |
| "message": f"missing required metadata fields: {', '.join(missing)}", | ||
| } | ||
|
|
||
| plugin_name = metadata["name"].strip() |
There was a problem hiding this comment.
If metadata["name"] is a numeric type (which can happen with some YAML parsers if the name is just numbers), calling .strip() directly will raise an AttributeError. Consider converting it to a string first.
| plugin_name = metadata["name"].strip() | |
| plugin_name = str(metadata["name"]).strip() |
| stdout = completed.stdout.strip() | ||
| if stdout: | ||
| try: | ||
| payload = json.loads(stdout) |
There was a problem hiding this comment.
json.loads(stdout) will fail if the worker process or the plugin being loaded prints any logs to stdout before the final JSON result. Since run_worker prints the JSON as the last line, consider parsing only the last line of stdout to be more robust against extra log output.
| payload = json.loads(stdout) | |
| payload = json.loads(stdout.splitlines()[-1]) |
|
Superseded by #716, which is opened directly from zouyonghe:main. |
1 participant
This change adds an MVP validation path for marketplace plugins so the repository can catch more than malformed
plugins.jsonentries or unreachable repository URLs.Today the collection only verifies index syntax and repository reachability. That lets broken plugins enter the market even when they are missing
metadata.yaml, lack a valid entrypoint, or fail during AstrBot's real plugin loading path. The result is that users can install plugins that immediately fail to import or initialize.The root cause is that marketplace validation stops before any plugin-structure or AstrBot-runtime checks happen. There was no reusable validator that clones a plugin repository, checks the expected files, and asks AstrBot's
PluginManager.load()whether the plugin can actually load.This PR adds that missing validation layer in two parts. First, it introduces
scripts/validate_plugins/run.py, which normalizes GitHub repository URLs, selects target plugins fromplugins.json, clones each repository, runs metadata and entrypoint prechecks, and then launches a subprocess worker that places the plugin into a temporary AstrBot root and executesPluginManager.load(specified_dir_name=...). Second, it adds.github/workflows/validate-plugin-smoke.yml, which runs this validator for changed plugin entries on pull requests and supports manualworkflow_dispatchruns for targeted checks or limited sweeps.The validator emits a structured JSON report with stage-specific failures such as
repo_url,clone,metadata,entrypoint,load,worker, andtimeout, so maintainers can tell why a plugin failed instead of only seeing a generic CI failure. The PR also includes focused unit tests for URL normalization, plugin selection, metadata validation, worker command construction, report aggregation, and worker output parsing.Validation used for this change:
python3 -m unittest tests.test_validate_plugins -vpython3 scripts/validate_plugins/run.py --helpruby -e 'require "yaml"; YAML.load_file(".github/workflows/validate-plugin-smoke.yml"); puts "workflow_ok"'Summary by Sourcery
Introduce an automated smoke validation pipeline for marketplace plugins that clones repositories, performs basic structural checks, attempts runtime loading via AstrBot, and reports structured results in CI.
New Features:
CI: